Graduation outcomes of 
students who entered 

New York City public Makin9 Connections 

schools in grade 5 or 6 as 
English learner students 


Michael J. Kieffer 

New York University 

Caroline E. Parker 

Education Development Center 


This study uses longitudinal administrative data from New York City public schools for 
2003/04 through 2012/13 to analyze high school graduation outcomes for students who 
entered New York City schools in grade 5 or 6 in 2003/04 as English learner students. 
Approximately 64 percent of students in these cohorts graduated from high school on 
time (within four years of entering grade 9), and an additional 15 percent graduated 
within six years of entering grade 9. Students in the two cohorts earned a variety of 
diploma types within six years, including the standard Regents diploma (41 percent), 
the less rigorous Local diploma (19 percent), and the more rigorous Advanced Regents 
diploma (19 percent). 


Why this study? 


A large and increasing number of students enter U.S. schools with limited proficiency in English (Bat- 
alova, Fix, & Murray, 2007). Some of these English learner students rapidly develop English proficiency 
and academic competence, while others make slower progress (August & Shanahan, 2006). Current 
assessment systems and crosS'Sectional studies provide information on the performance of students 
classified as English learner students at a particular point in time, but information on their long-term 
outcomes — particularly high school graduation rates and preparation for college — is scarce. Because the 
English learner classification is temporary, cross-sectional data reveal little about students who enter 
school as English learner students and either remain in that category for many years or are reclassified as 
former English learner students. 
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The large and diverse student population in New York City public schools offers a unique opportunity to 
study the graduation outcomes among students who enter school not yet proficient in English. To take 
advantage of this opportunity, Regional Educational Laboratory (REL) Northeast & Islands, in collabora- 
tion with its English Language Learners Alliance and the Research Alliance for New York City Schools 
at New York University, conducted this longitudinal study of the graduation outcomes of students who 
enter school as English learner students. It focuses on students who entered New York City public schools, 
either from another country or from another school system in the United States, in grades 5 and 6 in the 
2003/04 school year and were classified as English learner students. This population of students was exam- 
ined because there were sufficient years of data to determine how long they took to be reclassified as former 
English learner students and to determine whether they graduated on time or within six years. Students 
who entered in kindergarten-grade 4 before 2003/04 had sufficient years of data but were not included 
because they entered school before implementation of the No Child Left Behind Act, which substantially 
changed classification requirements for English learner students. 

As states and districts implement the federal Every Student Succeeds Act of 2015, they are encountering 
new opportunities to build and refine accountability systems to track their performance and that of individ- 
ual schools in meeting the needs of English learner students (see, for example, National Council of La Raza, 
2016). Improved reporting of high school graduation rates, in particular, may provide a more precise picture 
of English learner students’ successes and challenges. Researchers have argued that current approaches to 
tracking English learner students’ performance, which are based on the requirements of No Child Left 
Behind, are misleading and need to be revised (Hopkins, Thompson, Linquanti, Hakuta, & August, 2013). 
Specifically, high school graduation rates among English learner students are typically based on the outcomes 
of the subset of students designated as English learner students at the time of graduation or at the beginning 
of grade 9. The graduation rates for those students are consistently lower than overall high school graduation 
rates, as might be expected for students who have not met the criteria to be reclassified as former English 
learner students (Koelsch, 2009; Matthews & Mellom, 2012; Shneyderman, 2012; Stetser & Stillwell, 2014). 

New York State accountability report cards for 2010/11, which detail the academic performance of students, 
districts, and schools in the state, indicate a statewide four-year cohort graduation rate of 77 percent among 
all students and 52 percent among English learner students. More recent New York State accountability 
report cards indicate similar differences between these two groups in New York City’s 32 geographic dis- 
tricts (with some variation across districts; see http://data.nysed.gov). These state and regional district rates 
among English learner students are similar to national rates (Stetser & Stilwell, 2014). But these rates may 
be misleading because they do not account for students who entered school as English learner students 
prior to high school and who were subsequently reclassified as former English learner students (Hopkins 
et al., 2013). The current study addresses this concern by calculating graduation rates on the basis of all 
students who were ever English learner students, including both those who continued to be classified as 
English learner students and those who were later reclassified as former English learner students. Although 
the current study follows only students entering New York City public schools in grade 5 or 6, it provides an 
illustration for how this approach may be taken for students entering in other grades. 

In addition to describing the graduation outcomes of students who were ever English learner students, this 
study explores whether the outcomes differ for subgroups of students who gain English proficiency at differ- 
ent times and for those who were not reclassified within six years and were therefore considered long-term 
English learner students. Two studies of English learner students in California found that students who 
had not been reclassified within six years struggled academically. Although they functioned well socially in 
both their native language and English, they lacked academic vocabulary in English (Olsen, 2010) and, on 
average, had worse outcomes than did English learner students who had been reclassified in earlier grades 
(Hill, Weston, & Hayes, 2014). 
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A previous Regional Educational Laboratory Northeast & Islands study using longitudinal administrative 
data for 2003-12 found that most English learner students in New York City public schools took three to 
five years to reach standards for English proficiency (Kieffer & Parker, 2016). But a substantial proportion 
of students (24-44 percent, depending on the grade of entry) did not reach these standards after six or 
more years of schooling and thus became long-term English learner students. In particular, the percentage 
of students who became long-term English learner students was higher among students who entered New 
York City public schools in grade 5 or 6 (35 percent and 41 percent) than among students who entered in 
an earlier grade. The current study extends that work by exploring the high school graduation outcomes 
of this group of long-term English learner students and comparing their outcomes with those of short-term 
English learner students who entered in the same grades. 

Finally, in addition to describing graduation rates, the current study investigates the types of diplomas 
earned by students who were ever English learner students and also studies this outcome separately for 
short-term and long-term English learner students. Graduating students in New York City can earn one 
of several types of diploma, some of which signal stronger academic preparation than others. Students can 
earn a Regents diploma by completing 22 course credits and passing five New York State Regents exams 
(subject-specific tests designed to cover essential high school content and that serve collectively as a high 
school graduation requirement). Students can earn an Advanced Regents diploma by passing additional 
Regents exams (see appendix A). Until recently, students could also earn a New York City Local diploma 
by fulfilling the same 22-credit requirement, attaining the passing score of 65 on two to four Regents exams 
(depending on the year in which they first enrolled in grade 9), and attaining a nonpassing but adequate 
score (55-64) on the remaining Regents exams. This option ended for students who entered grade 9 after 
the 2007/08 school year (see appendix A). 

The type of diploma earned is a key indicator of students’ academic preparation for college. One study found 
that only 40 percent of New York City high school graduates with a Local diploma enrolled in college, 
compared with 60 percent of those with a Regents diploma and 80 percent of those with an Advanced 
Regents diploma (Coca, 2014). Furthermore, 20 percent of New York City high school graduates with a 
Local diploma who enrolled in college earned a degree within four years compared with 29 percent of those 
with a Regents diploma and 54 percent of those with an Advanced Regents diploma. Investigating the type 
of diploma earned by students who enter school as English learner students is thus vital to understanding 
their education trajectories. 


What the study examined 


This longitudinal study analyzes high school graduation outcomes of students who entered New York City 
public schools in grade 5 or 6 as English learner students. It extends the work of Kieffer and Parker (2016) 
by investigating the high school graduation rates and the types of diploma earned by the 1,734 students 
who entered New York City public schools (either from another country or from another school system in 
the United States) in grades 5 and 6 in the 2003/04 school year (the first year after implementation of the 
No Child Left Behind Act) and who were classified as English learner students, regardless of whether they 
were subsequently reclassified. 1 The study followed the two cohorts through two years after their expected 
graduation year (that is, through 2011/12 for the grade 5 cohort and 2012/13 for the grade 6 cohort; see box 
1 for information about the study methodology). 

The study explores differences between long-term and short-term English learner students. Of the 1,734 
students in the sample, 718 (41.4 percent) were identified as long-term English learner students and 1,016 
(58.6 percent) were identified as short-term English learner students (see appendix B for additional informa- 
tion about the study sample.) For this study long-term English learner students were defined as students who 
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Box 1. Data and methods 


This study used data compiled and processed by the Research Alliance for New York City Schools at New York 
University. 

Graduation rates were estimated using a method similar to the one used by the New York State Education 
Department and New York City Department of Education. Graduates are students who earned a Local diploma, 
Regents diploma, or Advanced Regents diploma (see appendix A for requirements for each diploma type). Non- 
graduates are students who dropped out of high school, remained enrolled in a New York City high school 
without graduating, or received either a General Educational Development certificate or an Individualized Edu- 
cation Program certificate, but nongraduates do not include students identified as transferring out of the New 
York City public school system to the school system in another jurisdiction. On-time graduation was defined as 
earning any type of diploma within four years of entering grade 9 for the first time. 

Cohort graduation rates include students who earned diplomas over the summer (for example, four-year 
cohort graduation rates count as graduates students who earned a diploma in the summer following the school 
year of their expected graduation). 

The study team used logistic regression to determine whether the differences in graduation outcomes 
between long-term and short-term English learner students were statistically significant. Logistic regression is 
used when the outcome variables are categorical (in this case, graduation and diploma type). To account for 
clustering of students within schools, robust standard errors were used; for students who attended multiple 
high schools within the period studied, school was specified as the most recent high school attended before 
graduating, continuing to attend without graduating, or dropping out. 

Additional logistic regression models were used to investigate whether five student background characteris- 
tics explained any of the differences between long-term and short-term English learner students: grade of entry 
(that is, grade 5 or 6), race/ethnicity, language spoken at home, eligibility for the federal school lunch program, 
and average income for the census tract in which the students’ home was located. All of these characteristics 
have previously been found to be associated with time to reclassification (Kieffer & Parker, 2016) and with 
graduation outcomes (Rumberger, 2001). For each research question the headings, figures, and text present 
the primary findings from the simple comparison without accounting for background characteristics; secondary 
findings incorporating background characteristics are presented in the main text where noted and in appendix B. 
See appendix B for additional information on methodology. 


were classified as English learner students when they entered school in grade 5 or 6 and were not reclassi- 
fled as former English learner students within six years (through grade 10 for those who entered in grade 5 
and through grade 11 for those who entered in grade 6), consistent with the New York City Department of 
Education (2013) definition. 2 Approximately 44 percent of long-term English learner students who entered 
in grade 5 and 21 percent who entered in grade 6 were reclassified as former English learner students before 
their expected year of high school graduation. Short-term English learner students were defined as those 
who were reclassified as former English learner students within six years. 

The study addressed three research questions: 

• What percentage of students who entered New York City public schools in grade 5 or 6 as English 
learner students graduated from high school on time (that is, within four years of entering grade 
9)? To what extent does the percentage differ between long-term and short-term English learner 
students? 

• What percentage of students who entered New York City public schools in grade 5 or 6 as English 
learner students graduated one or two years after their expected graduation year (that is, within 
five years or six years of entering grade 9)? To what extent does the percentage differ between long- 
term and short-term English learner students? 
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• What percentage of students who entered New York City public schools in grade 5 or 6 as English 
learner students received a New York City Local diploma, Regents diploma, or Advanced Regents 
diploma within six years of entering grade 9? To what extent does the percentage differ between 
long-term and short-term English learner students? 

What the study found 


This section presents the graduation rates for students who entered New York City public schools in grade 
5 or 6 as English learner students within four years, five years, or six years of entering grade 9. It also reports 
the types of diploma earned by these students. 

Approximately 64 percent of students who entered New York City public schools in grade 5 or 6 as English learner 
students graduated from high school on time 

The on-time cohort graduation rate was approximately 64 percent among all students who entered New 
York City public schools in grade 5 or 6 as English learner students (figure 1). This rate is 7 percentage 
points lower than the on-time graduation rate among all students in the New York City public school 
system. 

Graduation rates differed across subgroups that had different times to reclassification. Specifically, 55 percent 
of long-term English learner students graduated from high school on time, compared with 76 percent of 
short-term English learner students (see figure 1). The 21 percentage point difference was statistically sig- 
nificant ( p < .001). When student background characteristics were controlled for, the difference was similar 


Figure 1. The on-time cohort graduation rate was lower among students who were ever English 
learner students than among all students in New York City public schools, and the rate was lower 
among long-term English learner students than among short-term English learner students 
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a. Refers to students who entered New York City public schools in grade 5 or 6 during the 2003/04 school year as English learner 
students and were expected to graduate in 2009/10 or 2010/11 or were retained prior to grade 9 and thus expected to graduate 
later. 


b. Refers to all students in New York City public schools who were expected to graduate in 2009/10 or 2010/11 (that is, first-time 
grade 9 students in fall 2006 or fall 2007). 

Source: Authors’ analysis based on 2003/04-2012/13 data from the New York City Department of Education. 
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(20 percentage points; see table B3 in appendix B) and remained statistically significant ( p < .001). This 
indicates that the difference in on-time graduation rates between long-term and short-term English learner 
students is not explained by differences in their background characteristics. 

Approximately 15 percent of students who entered New York City public schools in grade 5 or 6 as English learner 
students graduated one or two years after their expected graduation year, yielding a six-year cohort graduation rate of 
79 percent 

Approximately 15 percent of students who were ever English learner students graduated one or two years 
after their expected graduation year (10 percent one year later and 5 percent two years later), yielding a 
cumulative six-year cohort graduation rate of 79.2 percent (figure 2). Some 8 percent of all students in New 
York City public schools graduated one or two years after their expected graduation year (7 percent one year 
later and 1 percent two years later), yielding a cumulative six-year cohort graduation rate of 79.1 percent (see 
figure 2). Thus, while students who were ever English learner students had a lower on-time graduation rate, 
their six-year graduation rate was similar to the rate among all students in New York City public schools. 

Some 19 percent of long-term English learner students and 12 percent of short-term English learner stu- 
dents graduated one or two years after their expected graduation year (figure 3). That is, a higher per- 
centage of long-term English learner students than of short-term English learner students stayed in school 
another year or two and successfully met graduation requirements. Nonetheless, the cumulative six-year 
graduation rate was lower among long-term English learner students (74 percent) than among short-term 
English learner students (87 percent). This difference was statistically significant ( p < .001). When student 
background characteristics were controlled for, the difference was similar (13 percentage points; see table 


Figure 2. While students who were ever English learner students had lower on-time and five-year 
graduation rates, their six-year graduation rate was similar to the rate among all students in New 
York City public schools 
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a. Refers to students who entered New York City public schools in grade 5 or 6 during the 2003/04 school year as English learner 
students and were expected to graduate in 2009/10 or 2010/11 or were retained prior to grade 9 and thus expected to graduate 
later. 


b. Refers to all students in New York City public schools who were expected to graduate in 2009/10 or 2010/11 (that is, first-time 
ninth grade students in fall 2006 or fall 2007). 

Source: Authors' analysis based on 2003/04-2012/13 data from the New York City Department of Education. 
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Figure 3. The four-year, five-year, and six-year graduation rates were lower among long-term 
English learner students than among short-term English learner students 


Graduation rate (percent) 



Note: Long-term and short-term English learner students refers to students who entered New York City public schools in grade 5 
or 6 during the 2003/04 school year as English learner students and were expected to graduate in 2009/10 or 2010/11 or were 
retained prior to grade 9 and thus expected to graduate later. 

Source: Authors’ analysis based on 2003/04-2012/13 data from the New York City Department of Education. 


B3 in appendix B) and remained statistically significant ( p < .001). This indicates that the difference in 
six-year graduation rates between long-term and short-term English learner students is not explained by 
differences in their background characteristics. 

Among students who entered New York City public schools in grade 5 or 6 as English learner students, approximately 
41 percent earned a Regents diploma, 19 percent earned a Local diploma, and 19 percent earned an Advanced 
Regents diploma 

The most common diploma type earned by students who were ever English learner students within six years 
of entering grade 9 was the Regents diploma (41 percent), followed by the Local diploma (19 percent) and 
the Advanced Regents diploma (19 percent; figure 4)- The most common diploma type among all students 
in New York City public schools was also the Regents diploma (42 percent), followed by the Advanced 
Regents diploma (20 percent) and the Local diploma (16 percent; see figure 4). 

The percentage of students who earned a Local diploma was higher among long-term English learner stu- 
dents (33 percent) than among short-term English learner students (11 percent), and the percentage of 
students who earned an Advanced Regents diploma was higher among short-term English learner students 
(31 percent) than among long-term English learner students (4 percent; figure 5). Both differences were sta- 
tistically significant ( p < .001). The percentage of students who earned a Regents diploma was also higher 
among short-term English learner students (45 percent) than among long-term English learner students 
(37 percent), a statistically significant difference ( p = .011). The percentage of students who did not earn a 
diploma was higher among long-term English learner students (26 percent) than among short-term English 
learner students (13 percent). Each of these differences was similar and remained statistically significant 
after student background characteristics were controlled for (all p-values < .001; see table B4 in appendix 
B). This indicates that the differences in diploma types earned by long-term and short-term English learner 
students are not explained by differences in their background characteristics. 
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Figure 4. The percentages of diploma types earned within six years of entering grade 9 were 
roughly similar among students who were ever English learner students and among all students in 
New York City public schools 
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a. Refers to students who entered New York City public schools in grade 5 or 6 during the 2003/04 school year as English learner 
students and were expected to graduate in 2009/10 or 2010/11 or were retained prior to grade 9 and thus expected to graduate 
later. 

b. Refers to all students in New York City public schools who were expected to graduate in 2009/10 or 2010/11 (that is, first-time 
ninth grade students in fall 2006 or fall 2007). 

Source: Authors’ analysis based on 2003/04-2012/13 data from the New York City Department of Education. 


Figure 5. The percentage of students who earned a Local diploma was higher among long-term 
English learner students than among short-term English learner students, while the percentage of 
students who earned an Advanced Regents diploma was higher among short-term English learner 
students 
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Note: Long-term and short-term English learner students refers to students who entered New York City public schools in grade 5 
or 6 during the 2003/04 school year as English learner students and were expected to graduate in 2009/10 or 2010/11 or were 
retained prior to grade 9 and thus expected to graduate later. 

Source: Authors’ analysis based on 2003/04-2012/13 data from the New York City Department of Education. 
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Implications of the study findings 


This study has implications both for understanding achievement patterns among English learner students 
more precisely by incorporating data on how English learner student status changes over time and account- 
ing for all students ever classified as English learner students, including those who later met standards for 
English proficiency. 

Longitudinal tracking of students who were ever English learner students provides important information about English 
learner students’ successes as well as challenges 

Because the Every Student Succeeds Act provides new flexibility to refine the accountability systems that 
track English learner students’ performance, states and districts may want to consider different approaches 
to capturing the successes and challenges of serving this population (for example, Hopkins et al, 2013, and 
National Council of La Raza, 2016). In the current study the on-time graduation rate for students who were 
ever English learner students (64 percent) is higher than might be assumed on the basis of prior estimates 
for students classified as English learner students at the time of high school graduation or grade 9 entry. For 
example, New York State recorded a 52 percent statewide graduation rate for English learner students in 
2010/11 (see http://data.nysed.gov), and similar rates were reported by Stetser & Stilwell (2014). Consistent 
with the study by Hopkins et al. (2013), the current study supports the potential value of using a stable, 
longitudinal definition of the English learner student population for accountability purposes. Future efforts 
to investigate the extent to which districts and schools are meeting the needs of English learner students 
may benefit from carefully tracking students over time, regardless of whether they have met standards for 
English proficiency. 

Six-year graduation rates among English learner students tell a different story than four-year graduation rates tell 

Approximately 15 percent of students who were ever English learner students did not graduate on time but 
did graduate within six years of entering grade 9. While the four-year graduation rate was approximately 
7 percentage points lower among ever English learner students than among all students in New York City 
public schools, their six-year graduation rates were within 0.1 percentage point of each other. This result 
suggests that six-year graduation rates may be particularly important to consider when evaluating the grad- 
uation outcomes of students who enter school as English learner students — and even more so when ana- 
lyzing outcomes among long-term English learner students, whose six-year graduation rate is 19 percentage 
points higher than their four-year graduation rate. Reporting of graduation rates has changed over time 
in both New York City and New York State. New York City includes six-year rates in its School Quality 
Snapshots, but New York State includes only four-year rates in its report cards. To appropriately capture 
the long-term attainment of students who enter school as English learner students, district, city, and state 
accountability efforts in New York and elsewhere could incorporate six-year cohort graduation rates. In 
addition, five- or six-year pathways to graduation, which are sometimes offered to provide flexibility for 
English learner students and other students, could be evaluated (Short & Fitzsimmons, 2007). 

English learner students earned a variety of diploma types, suggesting variation in their preparation for college 

By including all students who entered grade 5 or 6 as English learner students, this study provides import- 
ant information about the successes of the 60 percent who earned a Regents or Advanced Regents diploma. 
At the same time, 19 percent of ever English learner students — and 33 percent of long-term English learner 
students — earned a less rigorous Local diploma (compared with 16 percent of all students in New York City 
public schools). Because a smaller percentage of Local diploma holders go on to enroll in and persist in 
college (Coca, 2014), these results suggest that many students who entered as English learner students may 
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have graduated from high school inadequately prepared for college. Research is needed into the character- 
istics of the 33 percent of long-term English learner students who earned a Local diploma. Many are likely 
also to be students with disabilities, given the high proportion of long-term English learner students who 
are identified as having a disability (Kieffer & Parker, 2016) and the high number of students with disabili- 
ties who received Local diplomas in New York City (Isaacs, 2014). 

Limitations of the study 


The current findings are limited to students who entered New York City public schools in grade 5 or 6 
in the 2003/04 school year and cannot be generalized to English learner students who entered New York 
City schools before grade 5 and after grade 6. Although data were available for students who entered in 
earlier grades, those students were either classified as English learner students under the pre-No Child 
Left Behind criteria (that is, before 2003) or had not been followed long enough to calculate six-year grad- 
uation rates. Similarly, data were available for students who entered later than grade 6, but such students 
did not have enough time in New York City schools to be classified as long-term English learner students. 
Research using additional years of data is thus needed to investigate the graduation outcomes of students 
who entered school as English learner students in other grades. Approximately 65 percent of students who 
were ever English learner students between 2003/04 and 2010/11 entered between kindergarten and grade 
4 compared with 8.1 percent who entered in grades 5 or 6, suggesting that the students in this study may 
not be representative of the overall population in New York City public schools (Kieffer & Parker, in 2016). 

Although the descriptive and exploratory findings in the current study illustrate differences in outcomes 
between long-term and short-term English learner students, the data cannot establish the sources of these 
differences. The differences were not explained by differences in grade of entry, race/ethnicity, or socio- 
economic status, but the two groups may differ in other ways (for instance, in achievement, motivation, 
and education experiences) that cannot be taken into account in the correlational analyses. Thus this 
study does not indicate whether earlier reclassification causes higher or lower graduation rates but rather 
reveals a pattern that should be investigated further. Recent studies that have applied rigorous regression- 
discontinuity methods to evaluate the effects of reclassification on outcomes have found that reclassifi- 
cation has no effect or a negative effect on the outcomes of students who are close to the criteria for 
reclassification, which challenges the view that students should be reclassified earlier than they currently 
are in the districts and states studied (Robinson, 2011; Robinson-Cimpian & Thompson, 2015). However, 
Robinson-Cimpian and Thompson (2015) found evidence that reclassification has a positive effect in some 
districts, a negative effect in some districts, and no effect in others and concluded that the effect of reclas- 
sification should be investigated in the context of the services that are provided to current and former 
English learner students. 
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Appendix A. Diploma requirements in New York State 


The requirements of three New York State diplomas are relevant to the cohorts studied in this report. The 
requirements for the Regents diploma and Advanced Regents diploma remained the same during the years 
studied (table Al). The requirements for the Local diploma differed depending on the year of first entry 
into grade 9. To earn a Local diploma, students who entered in grade 6 and were not retained (that is, 
who entered grade 9 in fall 2006) were required to pass three Regents exams (scoring 65 or above) and to 
score in the 55-64 range on two additional Regents exams. Students in the sample who entered in grade 5 
and were not retained (that is, who entered grade 9 in fall 2007) were required to pass four Regents exams 
and score in the 55-64 range on one additional Regents exam. The option to receive a Local diploma was 
not available to students who entered grade 9 in fall 2008 or later, so students in the sample who entered 
in grade 5 and were retained one or more times prior to high school or who entered in grade 6 and were 
retained twice or more prior to high school were ineligible for this diploma. 


Table Al. Diploma requirements in New York State in the years studied 

1 Requirement 

Local diploma 

Regents diploma 

Advanced Regents diploma 

Units of credit 

22, including: 

4 English 

4 Social studies 

3 Science 

3 Math 

2 Physical education 

1 Foreign language 

1 Arts 

0.5 Health 

3.5 Electives 

22, including: 

4 English 

4 Social studies 

3 Science 

3 Math 

2 Physical education 

1 Foreign language 

1 Arts 

0.5 Health 

3.5 Electives 

22, including: 

4 English 

4 Social studies 

3 Science 

3 Math 

2 Physical education 

1 Foreign language 

1 Arts 

0.5 Health 

3.5 Electives 

Regents 

examinations 

Entering grade 9 in 2006: 

Passing score (65 or higher) on 
three exams; score of 55-64 on 

two exams 

Entering grade 9 in 2007: 

Passing score (65 or higher) on 
four exams; score of 55-64 on one 

exam 

All years studied: 

Passing score (65 or higher) on the 
English Language Arts exam, one 
math exam, the Global History and 
Geography exam, the U.S. History 
and Government exam, and one 

science exam 

All years studied: 

Passing score (65 or higher) on the 
English Language Arts exam, two 
or three math exams, the Global 
History and Geography exam, 
the U.S. History and Government 
exam, two science exams, and one 
foreign language exam 

Source: New York State Education Department (2007). 
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Appendix B. Data and methodology 


This appendix provides information on the data and methodology used to conduct this study. The first 
section describes the calculation of cohort graduation rates, which is important to understanding the 
sample sizes and attrition reported in the next section. Subsequent sections describe student characteristics, 
explain the logistic regression analyses, and display the odds ratios and fitted probabilities generated by the 
regression models. 

Cohort graduation rates 

The analyses in this study follow the method used by Kemple (2013) and Coca (2014) to estimate cohort 
graduation rates using longitudinal administrative data from New York City public schools. Graduates are 
students who earned a Local diploma, Regents diploma, or Advanced Regents diploma (see appendix A 
for requirements for each diploma type). Nongraduates are students who dropped out of high school, who 
remained enrolled in a New York City high school without graduating, or who received either a General 
Educational Development certificate or an Individualized Education Program certificate and exclude stu- 
dents identified as transferring out of the New York City public school system to the school system in 
another jurisdiction. 

Cohort graduation rates include students who earned diplomas over the summer in the relevant graduation 
rate (thus, the four-year cohort graduation rates count as graduates students who earned a diploma in the 
summer following the school year of their expected graduation). 

These definitions are consistent with the methods that the New York State Education Department and 
New York City Department of Education use to calculate graduation rates, with one exception. The analy- 
ses exclude students who transferred into a New York City public school after the 2003/04 school year 
(when the cohort entered grade 5 or 6 and the study’s dataset). In contrast, the New York State Education 
Department and New York City Department of Education include in their cohort graduation rates all 
students who transfer into a New York City high school after grade 9. The same approach was used to 
calculate the rates at which students earned each of the three diploma types, including the same definitions 
for nongraduates. 

Sample and attrition 

The original sample consisted of 2,851 students who entered New York City public schools in grade 5 or 
6 in 2003/04 and were initially classified as English learner students. Of these students 911 (32.0 percent) 
transferred from the New York City public school system to a school system in another jurisdiction prior to 
their expected graduation year (see tables B1 and B2 later in this appendix for descriptive information on 
these students); these students were not included in the analytic sample for calculating cohort graduation 
rates (see previous section), consistent with the method used by Kemple (2013) and Coca (2014) as well as 
in state- and national mandated reporting. 

Although it is conceivable that some students who dropped out could have been misclassified as transfers, 
the data were carefully audited by the state, and Kemple’s (2013) longitudinal investigation of trends in 
graduation and transfer rates found no evidence to support this suspicion. The decision to exclude transfers 
from the estimate of graduation rates is reinforced by Kemple (2013), who argues that making assumptions 
about whether transfer students graduated or dropped out would be inappropriate because no additional 
data on these students are available. Furthermore, this decision, like New York State’s decision to exclude 
transfers from its graduation calculation, “reflects an assumption that schools are not accountable for 
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students’ outcomes after they leave the system, as schools no longer have any influence over these students” 
(Kemple, 2013, p. 10). Students who dropped out of school were included in the sample — an inclusion 
essential for accurately calculating cohort graduation rates. 

An additional 9 students (0.3 percent of the original sample) had a missing value for the relevant gradu- 
ation variable, likely due to a clerical error, and were excluded from the analyses. Another 197 students 
(6.9 percent) could not be identified as either long-term or short-term English learner students, which may 
be due to their departure from New York City public schools before high school and their later return at 
some point in high school, such that they did not have six years in the public school system during which 
they could have been served as long-term English learner students. 

The analytic sample was thus defined as students who had the opportunity to become long-term English 
learner students by virtue of having been in New York City public schools for six or more years. After the 
exclusions just noted, the sample for analyses of on-time (four-year) cohort graduation outcomes consisted 
of 1,734 students. Because additional students transferred to another jurisdiction in each of the two years 
following their expected graduation year, the analytic samples were proportionately smaller for five-year 
graduation outcomes (1,638) and six-year graduation outcomes (1,576). 

Student characteristics 

Table B1 provides descriptive information on student characteristics for each of the analytic samples. To 
provide insight into the representativeness of the analytic samples, table B1 also provides descriptive infor- 
mation on student characteristics for all English learner students in grade 5 or 6 in 2003/04 (that is, includ- 
ing those excluded from the analytic samples because they had entered New York City public schools in an 
earlier grade) and for students excluded from the analytic samples because they transferred out of New York 
City schools at some point between 2003/04 and their expected graduation year. 

Race/ethnicity was represented with dichotomous variables for the four major categories used by New York 
City schools. Home language was represented with nine dichotomous variables representing the eight most 
prevalent home language groups in the city and an additional variable representing a category to include 
all of the smaller language groups. There were no missing data on the grade of entry, gender, race/ethnicity, 
or home language variables. Family income was represented with a dichotomous variable with a value of 1 
if students were eligible for free lunch under the federal school lunch program and 0 otherwise. Neighbor- 
hood income was represented with a continuous variable indicating the neighborhood average per capita 
income (as indexed by the average annual income in the census tract where the student resided). Family 
and neighborhood income were treated as time-invariant variables (that is, they were treated as constant 
over time). For many but not all students the earliest available value on which to base these variables 
corresponded to their year of entry to New York City public schools. For students who did not have values 
on these variables for their year of entry, data from later years were used. Neighborhood income was not 
available for 18 cases (1. 0-1.1 percent of each analytic sample); this absence was taken into account using 
a model-based approach in which a dummy variable represented absence and an arbitrary value was sub- 
stituted for the missing value. Including these together enabled the study to retain the complete analytic 
sample. Data on eligibility for the federal school lunch program were missing for 8 cases (0.5 percent of 
each analytic sample). The dummy variable method failed to retain these students in the sample because 
it perfectly predicted the dichotomous outcome of on-time graduation. Thus the value for these eight cases 
was set to eligible, the most reasonable value given the prevalence of students in the sample who participat- 
ed in the federal school lunch program. 
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Table Bl. Characteristics of students in the analytic samples; of all English learner students in 
grade 5 or 6 in 2003/04, regardless of their grade of entry; and of students who entered New 
York City public schools in grade 5 or 6 as English learner students in 2003/04 but transferred to 
another jurisdiction prior to their expected graduation year (percent unless otherwise indicated) 


Student characteristic 

Analytic sample: students 
who entered New York City 
public schools in grade 5 
or 6 in 2003/04 as English 
learner students with data on 
graduation outcomes after 

Four Five Six 

years years years 

All English learner students 
in grade 5 or 6 in 2003/04, 
regardless of grade of entry 

Students who entered New 
York City public schools in 
grade 5 or 6 in 2003/04 
as English learner students 
but transferred to another 
jurisdiction prior to their 
expected graduation year 

Total (number) 

1,734 

1,638 

1,576 

11,949 

911 

Grade cohort 

Entered in grade 5 

47.8 

45.5 

45.9 

22.7 

50.4 

Entered in grade 6 

52.3 

54.5 

54.1 

12.7 

49.6 

Gender 

Male 

51.4 

51.0 

50.6 

51.1 

52.7 

Race/ethnicity 

Asian and Pacific Islander 

22.3 

23.0 

23.4 

19.3 

19.0 

Black 

7.2 

7.1 

6.7 

6.4 

7.8 

Hispanic 

59.8 

58.7 

58.6 

64.8 

61.0 

White, non-Hispanic 

10.6 

11.1 

11.3 

9.0 

11.7 

Home language 

Spanish 

59.5 

58.4 

58.2 

64.6 

60.6 

Bengali 

1.7 

1.8 

1.8 

2.3 

0.5 

Chinese 

11.6 

12.1 

12.2 

8.7 

8.1 

Korean 

2.4 

2.6 

2.7 

1.9 

2.4 

Haitian Creole 

2.6 

2.6 

2.4 

2.5 

2.4 

French 

1.3 

1.4 

1.1 

1.3 

2.1 

Russian 

4.0 

4.1 

4.1 

2.7 

3.1 

Southeast Asian languages 

0.3 

0.4 

0.3 

0.5 

0.7 

Other 

16.5 

16.8 

17.1 

15.4 

20.1 

Socioeconomic indicators 

Eligible for free lunch under 






the federal school lunch 






program 

65.2 

65.1 

64.8 

73.1 

65.1 

Average neighborhood per 






capita income in the home 






census tract ($) 

20,415 

20,570 

20,550 

19,574 

21,799 


Source: Authors' analysis based on 2003/04-2012/13 data from the New York City Department of Education. 


Table B2 shows English language proficiency levels at the end of grade 5 or 6 for students in the three ana- 
lytic samples, for all English learner students in grade 5 or 6 in 2003/04, and for students excluded from the 
analytic sample because they transferred out of New York City schools. 

Logistic regression analyses 

To address the research questions, two logistic regression models were fitted for each outcome (graduation 
and diploma type at the expected graduation year, one year later, and two years later). For each outcome 
the first logistic regression model estimated the uncontrolled difference in the probability of the outcome 
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Table B2. English proficiency levels at the end of grade 5 or 6 of students in the analytic samples; 
of all English learner students in grade 5 or 6 in 2003/04, regardless of their grade of entry; 
and of students who entered New York City public schools in grade 5 or 6 as English learner 
students in 2003/04 but transferred to another jurisdiction prior to their expected graduation year 
(percent) 


English 

proficiency 

Analytic sample: students who 
entered New York City public 
schools in grade 5 or 6 in 2003/04 
as English learner students with 
data on graduation outcomes after 

Four years Five years Six years 

All English learner students 
in grade 5 or 6 in 2003/04, 
regardless of grade of entry 

Students who entered New York 
City public schools in grade 

5 or 6 in 2003/04 as English 
learner students but transferred 
to another jurisdiction prior to 
their expected graduation year 

Beginning 

46.7 46.1 45.7 

17.7 

44.8 


Intermediate 35.7 35.7 35.7 38.3 34.5 


Advanced 14.4 14.8 15.2 


34.5 


16.1 


Proficient 3.2 3.3 3.4 


9.6 


4.6 


Note: English proficiency levels are based on end-of-year rather than initial proficiency because data on initial proficiency are avail- 
able only for students who were newly entering New York City schools in the given grade. Initial English proficiency scores were 
also available only as continuous raw scores rather than as proficiency levels for the year studied. 

Source: Authors' analysis based on 2003/04-2012/13 data from the New York City Department of Education. 


between long-term and short-term English learner students. For instance, for the outcome of graduating 
with any diploma at the year of expected graduation, the model was 

logit p(ANYDIPLOMA Y0 ) = (l 0 + (l 1 LTELS 

where indicated the relationship between long-term English learner student status (with LTELS = 1 
for long-term English learner students and LTELS = 0 for short-term English learner students) and the 
logit probability of earning any diploma at the year of expected graduation (YO). The reference category 
was short-term English learner students, so the intercept ((3 0 ) indicated the logit probability for short-term 
English learner students. To interpret the magnitude of the fitted estimate of it was converted to an 
odds ratio, and to determine the statistical significance of j^, robust standard errors accounting for nesting 
within school and associated ^-statistics and p-values were estimated. 

For each outcome the second logistic regression model estimated the relationship between the probability 
of the outcome and long-term English learner student status, controlling for student characteristics. For 
example, for the outcome of graduating with any diploma at the year of expected graduation, the model was 

logit p(ANYDIPLOMA yo ) = (3 0 + j^LTELS + pfiRADEOFENTRY + ^MALE + Z/ACE/ETHNICITY + 
Z 5 HOM ELANGUAGE + p 6 SCHOOLLUNCHPROGRAM + fl 7 CTAVGINCOME 

where (J 2 (2 3 and (i 6 indicated the relationships between the logit probability of on-time graduation (by 
earning any diploma at the year of expected graduation) and the three dichotomous characteristics, Z 4 
and Z 5 indicated the relationships between the logit probability of on-time graduation and a set of dummy 
variables to indicate the categorical variables of race/ethnicity and home language, and (2 ? indicated the 
relationship between the logit probability of on-time graduation and the continuous variable indicating 
average income in the student’s home census tract, centered at the sample mean. The dichotomous vari- 
ables were specified such that the reference category referred to short-term English learner students who 
entered in grade 6; who were female, Hispanic, Spanish-speakers, eligible for the federal school lunch 
program; and who lived in a census tract with an average income at the sample average. Again, to interpret 
the magnitude of the fitted estimate of (Jj, it was converted in an odds ratio and to determine the statistical 
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significance of robust standard errors accounting for nesting within school and associated ^-statistics 
and p-values were estimated. 

The odds ratio and its statistical significance for the relationship between on-time graduation and long- 
term English learner status produced in the second controlled model were compared with those produced 
in the first uncontrolled model to evaluate the extent to which differences between these two groups were 
reduced when accounting for potentially confounding differences in student characteristics. 

Odds ratios and fitted probabilities from logistic regression models 

Tables B3 and B4 report the odds ratios and fitted probabilities from the logistic regression models. The 
odds ratio indicates the ratio of the odds that the outcome will occur for long-term English learner students 
to the odds that the outcome will occur for short-term English learner students. 


Table B3. Odds ratios and fitted probability of graduation within four years, five years, and six 
years of entering grade 9, without and with controls for student characteristics, among students 
who entered New York City public schools in grade 5 or 6 in 2003/04 as English learner students 


Estimate 

Four year rate 
(n = 1,734) 

Five-year rate 
(n = 1,638) 

Six year rate 
(n = 1,576) 

Without controls for student characteristics 

Odds ratio for long-term English learner students, 
relative to short-term English learner students 

0.36 (0.04) 

0.33 (0.04) 

0.35 (0.05) 

Long-term English learner students' fitted probability of graduation 

.54 

.67 

.74 

Short-term English learner students’ fitted probability of graduation 

.77 

.86 

.89 

With controls for student characteristics 

Odds ratio for long-term English learner students, 
relative to short-term English learner students 

0.39 (0.05) 

0.36 (0.05) 

0.38 (0.06) 

Long-term English learner students' fitted probability of graduation 3 

.57 

.70 

.76 

Short-term English learner students’ fitted probability of graduation 3 

.77 

.87 

.89 


Note: Numbers in parentheses are robust standard errors. Odds ratios indicate the ratio of the odds that the outcome will occur 
for long-term English learner students to the odds that the outcome will occur for short-term English learner students. 

a. Calculated using sample average values. 

Source: Authors' analysis based on 2003/04-2012/13 data form the New York City Department of Education. 
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Table B4. Odds ratios and fitted probability of earning an Advanced Regents diploma, a Regents 
diploma, or a Local diploma within six years of entering grade 9, without and with controlling for 
student characteristics, among students who entered New York City public schools in grade 5 or 6 
in 2003/04 as English learner students 


Estimate 

Six-year 
Local diploma 
(n = 1,576) 

Six-year 

Regents diploma 
(n = 1,576) 

Six-year 
Advanced 
Regents diploma 
(n = 1,576) 

Without controls for student characteristics 

Odds ratio for long-term English learner students, relative to short-term 
English learner students 

3.68 (0.59) 

0.70 (0.10) 

0.10 (0.03) 

Long-term English learner students’ fitted probability of graduation 

.33 

.37 

.04 

Short-term English learner students’ fitted probability of graduation 

.12 

.46 

.32 

With controls for student characteristics 

Odds ratio for long-term English learner students, relative to short-term 
English learner students 

3.31 (0.51) 

0.64 (0.09) 

0.10 (0.03) 

Long-term English learner students’ fitted probability of graduation 3 

.29 

.36 

.04 

Short-term English learner students’ fitted probability of graduation 3 

.11 

..46 

.26 


Note: Numbers in parentheses are robust standard errors in odds-ratios. Odds ratios indicate the ratio of the odds that the 
outcome will occur for long-term English learner students to the odds that the outcome will occur for short-term English learner 
students. 

a. Calculated using sample average values. 

Source: Authors' analysis based on New York City Department of Education data, 2003/04-2012/13. 
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Notes 


The authors wish to thank members of the English Language Learners Alliance, especially members of the 
advisory committee, for this study: Rosa Pietanza, Maite Sanchez, and Robert Measel. The advisory com- 
mittee provided valuable feedback on the design of this study and the interpretation of findings. 

1. Students who entered New York City schools after grade 6 were not included because they did not have 
enough years in school to be classified as long-term English learner students; after six years as English 
learner students, they reached their expected graduation date. Students who entered New York City 
schools in 2003/04 before grade 4 were not included because by 2012/13 they would not yet have been 
observed through the year of their expected graduation. Although data prior to 2003/04 are available, 
the processes for initially identifying English learner students were different prior to implementation of 
the No Child Left Behind Act, so findings that relied on those data would not be relevant to current 
questions about English learner students. The 1,734 students who entered New York City public schools 
in grades 5 and 6 account for 5.1 percent of English learner students across all New York City public 
schools in 2003/04- 

2. Long-term English learner students also included students who were retained in elementary or middle 
school as long as they had been classified as English learner students for six school years. Because they 
reached grade 10 or 11 later than did nonretained students, they may or may not have still been classi- 
fied as English learner students at that grade. 
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